Finding structure during incremental speech comprehension.

A core aspect of human speech comprehension is the ability to incrementally integrate consecutive words into a structured and coherent interpretation, aligning with the speaker's intended meaning. This rapid process is subject to multidimensional probabilistic constraints, including both linguistic knowledge and non-linguistic information within specific contexts, and it is their interpretative coherence that drives successful comprehension. To study the neural substrates of this process, we extract word-by-word measures of sentential structure from BERT, a deep language model, which effectively approximates the coherent outcomes of the dynamic interplay among various types of constraints. Using representational similarity analysis, we tested BERT parse depths and relevant corpus-based measures against the spatiotemporally resolved brain activity recorded by electro-/magnetoencephalography when participants were listening to the same sentences. Our results provide a detailed picture of the neurobiological processes involved in the incremental construction of structured interpretations. These findings show when and where coherent interpretations emerge through the evaluation and integration of multifaceted constraints in the brain, which engages bilateral brain regions extending beyond the classical fronto-temporal language system. Furthermore, this study provides empirical evidence supporting the use of artificial neural networks as computational models for revealing the neural dynamics underpinning complex cognitive processes in the brain.

Decoding the Real-Time Neurobiological Properties of Incremental Semantic Interpretation.

Communication through spoken language is a central human capacity, involving a wide range of complex computations that incrementally interpret each word into meaningful sentences. However, surprisingly little is known about the spatiotemporal properties of the complex neurobiological systems that support these dynamic predictive and integrative computations. Here, we focus on prediction, a core incremental processing operation guiding the interpretation of each upcoming word with respect to its preceding context. To investigate the neurobiological basis of how semantic constraints change and evolve as each word in a sentence accumulates over time, in a spoken sentence comprehension study, we analyzed the multivariate patterns of neural activity recorded by source-localized electro/magnetoencephalography (EMEG), using computational models capturing semantic constraints derived from the prior context on each upcoming word. Our results provide insights into predictive operations subserved by different regions within a bi-hemispheric system, which over time generate, refine, and evaluate constraints on each word as it is heard.

Cognitive Diversity in a Healthy Aging Cohort: Cross-Domain Cognition in the Cam-CAN Project.

Objective: Studies of "healthy" cognitive aging often focus on a limited set of measures that decline with age. The current study argues that defining and supporting healthy cognition requires understanding diverse cognitive performance across the lifespan. Method: Data from the Cambridge Centre for Aging and Neuroscience (Cam-CAN) cohort was examined across a range of cognitive domains. Performance was related to lifestyle including education, social engagement, and enrichment activities. Results: Results indicate variable relationships between cognition and age (positive, negative, or no relationship). Principal components analysis indicated maintained cognitive diversity across the adult lifespan, and that cognition-lifestyle relationships differed by age and domain. Discussion: Our findings support a view of normal cognitive aging as a lifelong developmental process with diverse relationships between cognition, lifestyle, and age. This reinforces the need for large-scale studies of cognitive aging to include a wider range of both ages and cognitive tasks.

Neural dynamics of semantic composition.

Human speech comprehension is remarkable for its immediacy and rapidity. The listener interprets an incrementally delivered auditory input, millisecond by millisecond as it is heard, in terms of complex multilevel representations of relevant linguistic and nonlinguistic knowledge. Central to this process are the neural computations involved in semantic combination, whereby the meanings of words are combined into more complex representations, as in the combination of a verb and its following direct object (DO) noun (e.g., "eat the apple"). These combinatorial processes form the backbone for incremental interpretation, enabling listeners to integrate the meaning of each word as it is heard into their dynamic interpretation of the current utterance. Focusing on the verb-DO noun relationship in simple spoken sentences, we applied multivariate pattern analysis and computational semantic modeling to source-localized electro/magnetoencephalographic data to map out the specific representational constraints that are constructed as each word is heard, and to determine how these constraints guide the interpretation of subsequent words in the utterance. Comparing context-independent semantic models of the DO noun with contextually constrained noun models reflecting the semantic properties of the preceding verb, we found that only the contextually constrained model showed a significant fit to the brain data. Pattern-based measures of directed connectivity across the left hemisphere language network revealed a continuous information flow among temporal, inferior frontal, and inferior parietal regions, underpinning the verb's modification of the DO noun's activated semantics. These results provide a plausible neural substrate for seamless real-time incremental interpretation on the observed millisecond time scales.

Balancing Prediction and Sensory Input in Speech Comprehension: The Spatiotemporal Dynamics of Word Recognition in Context.

Spoken word recognition in context is remarkably fast and accurate, with recognition times of ∼200 ms, typically well before the end of the word. The neurocomputational mechanisms underlying these contextual effects are still poorly understood. This study combines source-localized electroencephalographic and magnetoencephalographic (EMEG) measures of real-time brain activity with multivariate representational similarity analysis to determine directly the timing and computational content of the processes evoked as spoken words are heard in context, and to evaluate the respective roles of bottom-up and predictive processing mechanisms in the integration of sensory and contextual constraints. Male and female human participants heard simple (modifier-noun) English phrases that varied in the degree of semantic constraint that the modifier (W1) exerted on the noun (W2), as in pairs, such as "yellow banana." We used gating tasks to generate estimates of the probabilistic predictions generated by these constraints as well as measures of their interaction with the bottom-up perceptual input for W2. Representation similarity analysis models of these measures were tested against electroencephalographic and magnetoencephalographic brain data across a bilateral fronto-temporo-parietal language network. Consistent with probabilistic predictive processing accounts, we found early activation of semantic constraints in frontal cortex (LBA45) as W1 was heard. The effects of these constraints (at 100 ms after W2 onset in left middle temporal gyrus and at 140 ms in left Heschl's gyrus) were only detectable, however, after the initial phonemes of W2 had been heard. Within an overall predictive processing framework, bottom-up sensory inputs are still required to achieve early and robust spoken word recognition in context.SIGNIFICANCE STATEMENT Human listeners recognize spoken words in natural speech contexts with remarkable speed and accuracy, often identifying a word well before all of it has been heard. In this study, we investigate the brain systems that support this important capacity, using neuroimaging techniques that can track real-time brain activity during speech comprehension. This makes it possible to locate the brain areas that generate predictions about upcoming words and to show how these expectations are integrated with the evidence provided by the speech being heard. We use the timing and localization of these effects to provide the most specific account to date of how the brain achieves an optimal balance between prediction and sensory input in the interpretation of spoken language.

Entrainment to the CIECAM02 and CIELAB colour appearance models in the human cortex.

In human visual processing, information from the visual field passes through numerous transformations before perceptual attributes such as colour are derived. The sequence of transforms involved in constructing perceptions of colour can be approximated by colour appearance models such as the CIE (2002) colour appearance model, abbreviated as CIECAM02. In this study, we test the plausibility of CIECAM02 as a model of colour processing by looking for evidence of its cortical entrainment. The CIECAM02 model predicts that colour is split in to two opposing chromatic components, red-green and cyan-yellow (termed CIECAM02-a and CIECAM02-b respectively), and an achromatic component (termed CIECAM02-A). Entrainment of cortical activity to the outputs of these components was estimated using measurements of electro- and magnetoencephalographic (EMEG) activity, recorded while healthy subjects watched videos of dots changing colour. We find entrainment to chromatic component CIECAM02-a at approximately 35 ms latency bilaterally in occipital lobe regions, and entrainment to achromatic component CIECAM02-A at approximately 75 ms latency, also bilaterally in occipital regions. For comparison, transforms from a less physiologically plausible model (CIELAB) were also tested, with no significant entrainment found.

Relating dynamic brain states to dynamic machine states: Human and machine solutions to the speech recognition problem.

There is widespread interest in the relationship between the neurobiological systems supporting human cognition and emerging computational systems capable of emulating these capacities. Human speech comprehension, poorly understood as a neurobiological process, is an important case in point. Automatic Speech Recognition (ASR) systems with near-human levels of performance are now available, which provide a computationally explicit solution for the recognition of words in continuous speech. This research aims to bridge the gap between speech recognition processes in humans and machines, using novel multivariate techniques to compare incremental 'machine states', generated as the ASR analysis progresses over time, to the incremental 'brain states', measured using combined electro- and magneto-encephalography (EMEG), generated as the same inputs are heard by human listeners. This direct comparison of dynamic human and machine internal states, as they respond to the same incrementally delivered sensory input, revealed a significant correspondence between neural response patterns in human superior temporal cortex and the structural properties of ASR-derived phonetic models. Spatially coherent patches in human temporal cortex responded selectively to individual phonetic features defined on the basis of machine-extracted regularities in the speech to lexicon mapping process. These results demonstrate the feasibility of relating human and ASR solutions to the problem of speech recognition, and suggest the potential for further studies relating complex neural computations in human speech comprehension to the rapidly evolving ASR systems that address the same problem domain.

Syntactic Complexity and Frequency in the Neurocognitive Language System.

Prominent neurobiological models of language follow the widely accepted assumption that language comprehension requires two principal mechanisms: a lexicon storing the sound-to-meaning mapping of words, primarily involving bilateral temporal regions, and a combinatorial processor for syntactically structured items, such as phrases and sentences, localized in a left-lateralized network linking left inferior frontal gyrus (LIFG) and posterior temporal areas. However, recent research showing that the processing of simple phrasal sequences may engage only bilateral temporal areas, together with the claims of distributional approaches to grammar, raise the question of whether frequent phrases are stored alongside individual words in temporal areas. In this fMRI study, we varied the frequency of words and of short and long phrases in English. If frequent phrases are indeed stored, then only less frequent items should generate selective left frontotemporal activation, because memory traces for such items would be weaker or not available in temporal cortex. Complementary univariate and multivariate analyses revealed that, overall, simple words (verbs) and long phrases engaged LIFG and temporal areas, whereas short phrases engaged bilateral temporal areas, suggesting that syntactic complexity is a key factor for LIFG activation. Although we found a robust frequency effect for words in temporal areas, no frequency effects were found for the two phrasal conditions. These findings support the conclusion that long and short phrases are analyzed, respectively, in the left frontal network and in a bilateral temporal network but are not retrieved from memory in the same way as simple words during spoken language comprehension.

Conserved Sequence Processing in Primate Frontal Cortex.

An important aspect of animal perception and cognition is learning to recognize relationships between environmental events that predict others in time, a form of relational knowledge that can be assessed using sequence-learning paradigms. Humans are exquisitely sensitive to sequencing relationships, and their combinatorial capacities, most saliently in the domain of language, are unparalleled. Recent comparative research in human and nonhuman primates has obtained behavioral and neuroimaging evidence for evolutionarily conserved substrates involved in sequence processing. The findings carry implications for the origins of domain-general capacities underlying core language functions in humans. Here, we synthesize this research into a 'ventrodorsal gradient' model, where frontal cortex engagement along this axis depends on sequencing complexity, mapping onto the sequencing capacities of different species.

Tonotopic representation of loudness in the human cortex.

A prominent feature of the auditory system is that neurons show tuning to audio frequency; each neuron has a characteristic frequency (CF) to which it is most sensitive. Furthermore, there is an orderly mapping of CF to position, which is called tonotopic organization and which is observed at many levels of the auditory system. In a previous study (Thwaites et al., 2016) we examined cortical entrainment to two auditory transforms predicted by a model of loudness, instantaneous loudness and short-term loudness, using speech as the input signal. The model is based on the assumption that neural activity is combined across CFs (i.e. across frequency channels) before the transform to short-term loudness. However, it is also possible that short-term loudness is determined on a channel-specific basis. Here we tested these possibilities by assessing neural entrainment to the overall and channel-specific instantaneous loudness and the overall and channel-specific short-term loudness. The results showed entrainment to channel-specific instantaneous loudness at latencies of 45 and 100 ms (bilaterally, in and around Heschl's gyrus). There was entrainment to overall instantaneous loudness at 165 ms in dorso-lateral sulcus (DLS). Entrainment to overall short-term loudness occurred primarily at 275 ms, bilaterally in DLS and superior temporal sulcus. There was only weak evidence for entrainment to channel-specific short-term loudness.

Domain-specific and Domain-general Processing in Left Perisylvian Cortex: Evidence from Russian.

The processing of words containing inflectional affixes triggers morphophonological parsing and affix-related grammatical information processing. Increased perceptual complexity related to stem-affix parsing is hypothesized to create predominantly domain-general processing demands, whereas grammatical processing primarily implicates domain-specific linguistic demands. Exploiting the properties of Russian morphology and syntax, we designed an fMRI experiment to separate out the neural systems supporting these two demand types, contrasting inflectional complexity, syntactic (phrasal) complexity, and derivational complexity in three comparisons: (a) increase in parsing demands while controlling for grammatical complexity (inflections vs. phrases), (b) increase in grammatical processing demands, and (c) combined demands of morphophonological parsing and grammatical processing (inflections and phrases vs. derivations). Left inferior frontal and bilateral temporal areas are most active when the two demand types are combined, with inflectional and phrasal complexity contrasting strongly with derivational complexity (which generated only bilateral temporal activity). Increased stem-affix parsing demands alone did not produce unique activations, whereas grammatical structure processing activated bilateral superior and middle temporal areas. Selective left frontotemporal language system engagement for short phrases and inflections seems to be driven by simultaneous and interdependent domain-general and domain-specific processing demands.

Representation of Instantaneous and Short-Term Loudness in the Human Cortex.

Acoustic signals pass through numerous transforms in the auditory system before perceptual attributes such as loudness and pitch are derived. However, relatively little is known as to exactly when these transformations happen, and where, cortically or sub-cortically, they occur. In an effort to examine this, we investigated the latencies and locations of cortical entrainment to two transforms predicted by a model of loudness perception for time-varying sounds: the transforms were instantaneous loudness and short-term loudness, where the latter is hypothesized to be derived from the former and therefore should occur later in time. Entrainment of cortical activity was estimated from electro- and magneto-encephalographic (EMEG) activity, recorded while healthy subjects listened to continuous speech. There was entrainment to instantaneous loudness bilaterally at 45, 100, and 165 ms, in Heschl's gyrus, dorsal lateral sulcus, and Heschl's gyrus, respectively. Entrainment to short-term loudness was found in both the dorsal lateral sulcus and superior temporal sulcus at 275 ms. These results suggest that short-term loudness is derived from instantaneous loudness, and that this derivation occurs after processing in sub-cortical structures.

Structure, form, and meaning in the mental lexicon: evidence from Arabic.

Does the organization of the mental lexicon reflect the combination of abstract underlying morphemic units or the concatenation of word-level phonological units? We address these fundamental issues in Arabic, a Semitic language where every surface form is potentially analyzable into abstract morphemic units - the word pattern and the root - and where this view contrasts with stem-based approaches, chiefly driven by linguistic considerations, in which neither roots nor word patterns play independent roles in word formation and lexical representation. Five cross-modal priming experiments examine the processing of morphologically complex forms in the three major subdivisions of the Arabic lexicon - deverbal nouns, verbs, and primitive nouns. The results demonstrate that root and word pattern morphemes function as abstract cognitive entities, operating independently of semantic factors and dissociable from possible phonological confounds, while stem-based approaches consistently fail to accommodate the basic psycholinguistic properties of the Arabic mental lexicon.

Auditory sequence processing reveals evolutionarily conserved regions of frontal cortex in macaques and humans.

An evolutionary account of human language as a neurobiological system must distinguish between human-unique neurocognitive processes supporting language and evolutionarily conserved, domain-general processes that can be traced back to our primate ancestors. Neuroimaging studies across species may determine whether candidate neural processes are supported by homologous, functionally conserved brain areas or by different neurobiological substrates. Here we use functional magnetic resonance imaging in Rhesus macaques and humans to examine the brain regions involved in processing the ordering relationships between auditory nonsense words in rule-based sequences. We find that key regions in the human ventral frontal and opercular cortex have functional counterparts in the monkey brain. These regions are also known to be associated with initial stages of human syntactic processing. This study raises the possibility that certain ventral frontal neural systems, which play a significant role in language function in modern humans, originally evolved to support domain-general abilities involved in sequence processing.

Tracking cortical entrainment in neural activity: auditory processes in human temporal cortex.

A primary objective for cognitive neuroscience is to identify how features of the sensory environment are encoded in neural activity. Current auditory models of loudness perception can be used to make detailed predictions about the neural activity of the cortex as an individual listens to speech. We used two such models (loudness-sones and loudness-phons), varying in their psychophysiological realism, to predict the instantaneous loudness contours produced by 480 isolated words. These two sets of 480 contours were used to search for electrophysiological evidence of loudness processing in whole-brain recordings of electro- and magneto-encephalographic (EMEG) activity, recorded while subjects listened to the words. The technique identified a bilateral sequence of loudness processes, predicted by the more realistic loudness-sones model, that begin in auditory cortex at ~80 ms and subsequently reappear, tracking progressively down the superior temporal sulcus (STS) at lags from 230 to 330 ms. The technique was then extended to search for regions sensitive to the fundamental frequency (F0) of the voiced parts of the speech. It identified a bilateral F0 process in auditory cortex at a lag of ~90 ms, which was not followed by activity in STS. The results suggest that loudness information is being used to guide the analysis of the speech stream as it proceeds beyond auditory cortex down STS toward the temporal pole.

Brain Network Connectivity During Language Comprehension: Interacting Linguistic and Perceptual Subsystems.

The dynamic neural processes underlying spoken language comprehension require the real-time integration of general perceptual and specialized linguistic information. We recorded combined electro- and magnetoencephalographic measurements of participants listening to spoken words varying in perceptual and linguistic complexity. Combinatorial linguistic complexity processing was consistently localized to left perisylvian cortices, whereas competition-based perceptual complexity triggered distributed activity over both hemispheres. Functional connectivity showed that linguistically complex words engaged a distributed network of oscillations in the gamma band (20-60 Hz), which only partially overlapped with the network supporting perceptual analysis. Both processes enhanced cross-talk between left temporal regions and bilateral pars orbitalis (BA47). The left-lateralized synchrony between temporal regions and pars opercularis (BA44) was specific to the linguistically complex words, suggesting a specific role of left frontotemporal cross-cortical interactions in morphosyntactic computations. Synchronizations in oscillatory dynamics reveal the transient coupling of functional networks that support specific computational processes in language comprehension.

Grammatical analysis as a distributed neurobiological function.

Language processing engages large-scale functional networks in both hemispheres. Although it is widely accepted that left perisylvian regions have a key role in supporting complex grammatical computations, patient data suggest that some aspects of grammatical processing could be supported bilaterally. We investigated the distribution and the nature of grammatical computations across language processing networks by comparing two types of combinatorial grammatical sequences--inflectionally complex words and minimal phrases--and contrasting them with grammatically simple words. Novel multivariate analyses revealed that they engage a coalition of separable subsystems: inflected forms triggered left-lateralized activation, dissociable into dorsal processes supporting morphophonological parsing and ventral, lexically driven morphosyntactic processes. In contrast, simple phrases activated a consistently bilateral pattern of temporal regions, overlapping with inflectional activations in L middle temporal gyrus. These data confirm the role of the left-lateralized frontotemporal network in supporting complex grammatical computations. Critically, they also point to the capacity of bilateral temporal regions to support simple, linear grammatical computations. This is consistent with a dual neurobiological framework where phylogenetically older bihemispheric systems form part of the network that supports language function in the modern human, and where significant capacities for language comprehension remain intact even following severe left hemisphere damage.

The Cambridge Centre for Ageing and Neuroscience (Cam-CAN) study protocol: a cross-sectional, lifespan, multidisciplinary examination of healthy cognitive ageing.

BACKGROUND: As greater numbers of us are living longer, it is increasingly important to understand how we can age healthily. Although old age is often stereotyped as a time of declining mental abilities and inflexibility, cognitive neuroscience reveals that older adults use neural and cognitive resources flexibly, recruiting novel neural regions and cognitive processes when necessary. Our aim in this project is to understand how age-related changes to neural structure and function interact to support cognitive abilities across the lifespan. METHODS/DESIGN: We are recruiting a population-based cohort of 3000 adults aged 18 and over into Stage 1 of the project, where they complete an interview including health and lifestyle questions, a core cognitive assessment, and a self-completed questionnaire of lifetime experiences and physical activity. Of those interviewed, 700 participants aged 18-87 (100 per age decile) continue to Stage 2 where they undergo cognitive testing and provide measures of brain structure and function. Cognition is assessed across multiple domains including attention and executive control, language, memory, emotion, action control and learning. A subset of 280 adults return for in-depth neurocognitive assessment in Stage 3, using functional neuroimaging experiments across our key cognitive domains.Formal statistical models will be used to examine the changes that occur with healthy ageing, and to evaluate age-related reorganisation in terms of cognitive and neural functions invoked to compensate for overall age-related brain structural decline. Taken together the three stages provide deep phenotyping that will allow us to measure neural activity and flexibility during performance across a number of core cognitive functions. This approach offers hypothesis-driven insights into the relationship between brain and behaviour in healthy ageing that are relevant to the general population. DISCUSSION: Our study is a unique resource of neuroimaging and cognitive measures relevant to change across the adult lifespan. Because we focus on normal age-related changes, our results may contribute to changing views about the ageing process, lead to targeted interventions, and reveal how normal ageing relates to frail ageing in clinicopathological conditions such as Alzheimer's disease.

Optimally efficient neural systems for processing spoken language.

Cognitive models claim that spoken words are recognized by an optimally efficient sequential analysis process. Evidence for this is the finding that nonwords are recognized as soon as they deviate from all real words (Marslen-Wilson 1984), reflecting continuous evaluation of speech inputs against lexical representations. Here, we investigate the brain mechanisms supporting this core aspect of word recognition and examine the processes of competition and selection among multiple word candidates. Based on new behavioral support for optimal efficiency in lexical access from speech, a functional magnetic resonance imaging study showed that words with later nonword points generated increased activation in the left superior and middle temporal gyrus (Brodmann area [BA] 21/22), implicating these regions in dynamic sound-meaning mapping. We investigated competition and selection by manipulating the number of initially activated word candidates (competition) and their later drop-out rate (selection). Increased lexical competition enhanced activity in bilateral ventral inferior frontal gyrus (BA 47/45), while increased lexical selection demands activated bilateral dorsal inferior frontal gyrus (BA 44/45). These findings indicate functional differentiation of the fronto-temporal systems for processing spoken language, with left middle temporal gyrus (MTG) and superior temporal gyrus (STG) involved in mapping sounds to meaning, bilateral ventral inferior frontal gyrus (IFG) engaged in less constrained early competition processing, and bilateral dorsal IFG engaged in later, more fine-grained selection processes.